Delta Assertion

When using a Shared Fixture (page X) such as a test database, it can be hard to code the assertions that state what the content of the fixture should be after the system under test (SUT) has been exercised because other tests may have created objects in the fixture that our assertions may detect and which may cause our assertions to fail. One solution is to isolate this test from all other tests by using a Database Partitioning Scheme (see Database Sandbox on page X) but what can we do if this option is not available to us?

Using Delta Assertions allows us to be less dependent on what data already exist in the Shared Fixture.

How It Works

Before exercising the SUT, we take a snapshot of relevant parts of the Shared Fixture. After exercising the SUT, we specify our assertions relative to the saved snapshot. The Delta Assertions typically verify that the number of objects has changed by the right number and the contents of collections of objects have been augmented by the expected objects.

When To Use It

We can use a Delta Assertion whenever we don't have full control over the test fixture and we want to avoid Interacting Tests (see Erratic Test on page X). Using Delta Assertions will help us make our tests more resilient to changes in the fixture. We can also use Delta Assertions in concert with Implicit Teardown (page X) to detect memory or data leaks in the code that we are testing. See the sidebar Using Delta Assertions to Detect Data Leakage (page X)
Include the sidebar 'Using Delta Assertions to Detect Data Leakage' on opposite page.
for a more detailed description.

Delta Assertions work well when tests run one after another from the same Test Runner (page X) but they cannot prevent a Test Run War (see Erratic Test) because these are caused by tests being run at the same time from different processes. Delta Assertion works whenever the state of the SUT and the fixture are only being modified by our own test. If other tests are running in parallel (not before or after but at the same time), a Delta Assertion won't be sufficient to avoid the problem.

Implementation Notes

When saving the pre-test state of the Shared Fixture or SUT, we must make sure that the SUT cannot change our snapshot. For example, if we save a collection of objects returned by the SUT, we must ensure we do a deep copy; a shallow copy only copies the Collection object and not the objects it refers. That would allow the SUT to modify the objects it returned to us before we exercised it and we would lose the reference snapshot with which we are comparing the after state.

We can ensure that we have the right result several different ways. Assuming that our test adds any new objects it plans to modify, one approach is to first check that the result collection 1) has the right number of items, 2) contains all the pre-test items, and 3) contains the new Expected Objects (see State Verification on page X). The other approach is to remove all the saved items from the result collection and then compare what remains with the collection of new expected objects. Both of these approaches can be hidden behind a Custom Assertion (page X) or Verification Method (see Custom Assertion).

Motivating Example

The following test retrieves some objects from the SUT. It then compares the objects if found with those it expected.

   public void testGetFlightsByOriginAirport_OneOutboundFlight() throws Exception {
      FlightDto expectedFlightDto = createNewFlightBetweenExistingAirports();
      // Exercise System
      facade.createFlight( expectedFlightDto.getOriginAirportId(),
                 expectedFlightDto.getDestinationAirportId());
      // Verify Outcome
      List flightsAtOrigin = facade.getFlightsByOriginAirport(
                       expectedFlightDto.getOriginAirportId());
      assertOnly1FlightInDtoList( "Outbound flight at origin", expectedFlightDto,
                                  flightsAtOrigin);
   }
Example SharedFixtureNaiveAssertion embedded from java/com/clrstream/ex6/services/test/SharedFixtureFlightManagementFacadeTest.java

Unfortunately, since this is a Shared Fixture, other tests that ran before us may have added objects as well and that could cause this test to fail because we have additional objects that we don't expect.

Refactoring Notes

To convert to using a Delta Assertion, we must first take a snapshot of the data (or collection of objects) we will later be asserting on. Next, we need to modify our assertions to focus on the difference between the pre-test data/objects and the post-test data/objects. To avoid introducing Conditional Test Logic (page X) into the Test Method (page X), we may want to introduce a new Custom Assertion. We may be able to use existing assertions (custom or otherwise) as a starting point but we'll probably have to modify them to take into account the pre-test data.

Example: Delta Assertion

In this version of the test, we are using a Delta Assertion to verify objects added when we exercised the SUT. Note that we are verifying that we have one more object than before and that the collection of objects returned by the SUT includes the new Expected Object and all the objects that it previously contained.

   public void testCreateFlight_Delta() throws Exception {
      FlightDto expectedFlightDto = createNewFlightBetweenExistingAirports();
      // remember prior state
      List flightsBeforeCreate = facade.getFlightsByOriginAirport( expectedFlightDto.getOriginAirportId());
      // Exercise System
      facade.createFlight( expectedFlightDto.getOriginAirportId(),
                    expectedFlightDto.getDestinationAirportId());
      // Verify Outcome relative to prior state
      List flightsAfterCreate = facade.getFlightsByOriginAirport( expectedFlightDto.getOriginAirportId());
      assertFlightIncludedInDtoList( "new flight ", expectedFlightDto,
                                     flightsAfterCreate);
      assertAllFlightsIncludedInDtoList( "previous flights", flightsBeforeCreate,
                                         flightsAfterCreate);
      assertEquals( "Number of flights after create", flightsBeforeCreate.size()+1,
                    flightsAfterCreate.size());
   }
Example SharedFixtureDeltaAssertion embedded from java/com/clrstream/ex6/services/test/SharedFixtureFlightManagementFacadeTest.java

Since the SUT is returning Data Transfer Objects[CJ2EEP] (DTOs), we can be assured that the objects we saved before exercising the SUT cannot possibly change. We've modified our Custom Assertions to ignore the pre-test objects (by not insisting that the Expected Object is the only one) and written a new one that ensures that all the pre-test objects are also present. Another way to do this is to remove the pre-test objects from the result collection and then verify that only the new Expected Object(s) are left.

I've omitted the implementation of the Custom Assertions as they are purely an exercise in comparing objects and are not salient to understanding the Delta Assertion pattern. The "test infected" amongst us, would of course write the Custom Assertions driven by some Custom Assertion Tests (see Custom Assertion).

Page generated at Wed Feb 09 16:39:37 +1100 2011

Copyright © 2003-2008 Gerard Meszaros all rights reserved